fix: sync through empty epochs in range sync#9417
Conversation
A peer's correct response for an epoch in which every slot was skipped is zero blocks. Range sync rejected that response, so the node could not sync past an empty epoch during periods of poor chain liveness. Two interacting defects, both introduced during the fulu/PeerDAS sync work: 1. validateBlockByRangeResponse threw MISSING_BLOCKS_RESPONSE on a 0-block response. In sendBatch this became a downloadingError; since every peer returns empty for a genuinely empty epoch, all retries failed and after MAX_BATCH_DOWNLOAD_ATTEMPTS the batch threw MAX_DOWNLOAD_ATTEMPTS and the whole SyncChain errored out. A second copy of the throw in validateResponses fired for post-Deneb epochs that also carry a blobs/columns request. 2. The MAX_LOOK_AHEAD_EPOCHS guard in includeNextBatch counted batches in the AwaitingValidation state. Empty batches reach AwaitingValidation but never call advanceChain (only non-empty batches do), so lastEpochWithProcessBlocks never advances. A run of empty epochs longer than MAX_LOOK_AHEAD_EPOCHS (2) filled the look-ahead window, includeNextBatch returned null, and the chain stalled with nothing left to download or process. This deadlock is why the naive "return [] instead of throw" fix had previously been reverted. Archeology: - downloadByRange.ts was introduced in #8200 (feat: refactor block input, 2025-09-18) -- the fulu block-input/PeerDAS sync rewrite that deleted the old Deneb-era beaconBlocksMaybeBlobsByRange.ts downloader, which had tolerated empty epochs. The empty-blocks throw was then cemented in #8482 (fix: refactor validateColumnsByRangeResponse, 2025-10-27, b992c32). - The MAX_LOOK_AHEAD_EPOCHS guard was added 2025-08-11 (38889e2) during PeerDAS sync work. The original 2021 BATCH_BUFFER_SIZE check directly above it (f46b63f, dapplion) deliberately excluded AwaitingValidation batches "to prevent stalling sync if the current processing window is contained in a long range of skip slots." The new guard did not replicate that carve-out, re-creating the exact stall the 2021 code was written to avoid. Fix: - validateBlockByRangeResponse returns an empty result with a MISSING_BLOCKS_RESPONSE warning instead of throwing. Whether an epoch is genuinely empty is enforced at the chain level: the empty batch is held in AwaitingValidation and only confirmed once a later batch imports a block, so a peer cannot stall sync by falsely claiming an epoch is empty. - validateResponses skips data-sidecar validation when there are no blocks in the data request's slot range (parent-by-root columns and envelopes still run). - includeNextBatch's look-ahead guard ignores AwaitingValidation batches, mirroring the BATCH_BUFFER_SIZE carve-out. Tests: - validateBlockByRangeResponse accepts an empty response during chain liveness issues. - validateResponses accepts an empty epoch that carries a data request. - chain.test.ts gains a "3 epochs of skipped slots" case -- a run longer than MAX_LOOK_AHEAD_EPOCHS -- which previously deadlocked. Refs: #8147
validateResponses had three separate "nothing to validate here, fall through to parent-by-root columns + envelope validation" paths, with that parent + envelope tail duplicated between the no-data-request branch and the main path. The empty-epoch fix made the duplication more pronounced by adding a third skip path. Collapse them: gate getBlocksForDataValidation on `dataRequest` (defaulting to an empty list when absent) and let the single shared tail handle parent-by-root columns and envelopes for every case. This deletes the no-data-request early return entirely, and flattens data-sidecar validation from an outer `if (blocksForDataValidation.length > 0)` wrapper into per-request guards. Behavior-preserving: an envelopes-only / parent-payload-only request and an empty epoch now flow through the same path that already validated them, minus the duplicated branch. Net -20 lines. Adds a characterization test pinning the no-data-request (envelopes-only) path, which previously had no direct unit coverage.
There was a problem hiding this comment.
Code Review
This pull request addresses sync deadlocks during periods of poor chain liveness by allowing the sync chain to process empty epochs (epochs with zero blocks) without stalling or throwing errors. It updates the look-ahead window check to only count pending batches, preventing empty batches in AwaitingValidation from filling the window. Additionally, it refactors the response validation to return warnings instead of throwing errors when zero blocks are returned for an epoch, and adds corresponding regression tests. A review comment suggests simplifying a redundant ternary operator in validateResponses when passing validated blocks to getBlocksForDataValidation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe749a76a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
| Benchmark suite | Current: 443e8ce | Previous: 2e689b4 | Ratio |
|---|---|---|---|
| send data - 1000 16384B messages | 71.664 ms/op | 22.447 ms/op | 3.19 |
Full benchmark results
| Benchmark suite | Current: 443e8ce | Previous: 2e689b4 | Ratio |
|---|---|---|---|
| getPubkeys - index2pubkey - req 1000 vs - 250000 vc | 1.2718 ms/op | 1.0785 ms/op | 1.18 |
| getPubkeys - validatorsArr - req 1000 vs - 250000 vc | 40.552 us/op | 43.636 us/op | 0.93 |
| BLS verify - blst | 723.50 us/op | 654.89 us/op | 1.10 |
| BLS verifyMultipleSignatures 3 - blst | 1.3745 ms/op | 1.4150 ms/op | 0.97 |
| BLS verifyMultipleSignatures 8 - blst | 2.1820 ms/op | 2.4208 ms/op | 0.90 |
| BLS verifyMultipleSignatures 32 - blst | 6.8902 ms/op | 6.9483 ms/op | 0.99 |
| BLS verifyMultipleSignatures 64 - blst | 13.152 ms/op | 13.124 ms/op | 1.00 |
| BLS verifyMultipleSignatures 128 - blst | 25.853 ms/op | 25.597 ms/op | 1.01 |
| BLS deserializing 10000 signatures | 650.62 ms/op | 648.43 ms/op | 1.00 |
| BLS deserializing 100000 signatures | 6.5731 s/op | 6.4719 s/op | 1.02 |
| BLS verifyMultipleSignatures - same message - 3 - blst | 823.01 us/op | 770.43 us/op | 1.07 |
| BLS verifyMultipleSignatures - same message - 8 - blst | 967.51 us/op | 899.24 us/op | 1.08 |
| BLS verifyMultipleSignatures - same message - 32 - blst | 1.5360 ms/op | 1.4809 ms/op | 1.04 |
| BLS verifyMultipleSignatures - same message - 64 - blst | 2.3849 ms/op | 2.3849 ms/op | 1.00 |
| BLS verifyMultipleSignatures - same message - 128 - blst | 4.0682 ms/op | 4.1197 ms/op | 0.99 |
| BLS aggregatePubkeys 32 - blst | 17.871 us/op | 18.066 us/op | 0.99 |
| BLS aggregatePubkeys 128 - blst | 63.141 us/op | 64.395 us/op | 0.98 |
| getSlashingsAndExits - default max | 50.545 us/op | 61.716 us/op | 0.82 |
| getSlashingsAndExits - 2k | 360.14 us/op | 409.46 us/op | 0.88 |
| proposeBlockBody type=full, size=empty | 1.0103 ms/op | 874.92 us/op | 1.15 |
| isKnown best case - 1 super set check | 191.00 ns/op | 172.00 ns/op | 1.11 |
| isKnown normal case - 2 super set checks | 165.00 ns/op | 169.00 ns/op | 0.98 |
| isKnown worse case - 16 super set checks | 167.00 ns/op | 170.00 ns/op | 0.98 |
| validate api signedAggregateAndProof - struct | 1.5379 ms/op | 1.5184 ms/op | 1.01 |
| validate gossip signedAggregateAndProof - struct | 1.5315 ms/op | 1.5206 ms/op | 1.01 |
| batch validate gossip attestation - vc 640000 - chunk 32 | 109.61 us/op | 109.27 us/op | 1.00 |
| batch validate gossip attestation - vc 640000 - chunk 64 | 95.525 us/op | 99.336 us/op | 0.96 |
| batch validate gossip attestation - vc 640000 - chunk 128 | 90.638 us/op | 92.500 us/op | 0.98 |
| batch validate gossip attestation - vc 640000 - chunk 256 | 87.729 us/op | 89.211 us/op | 0.98 |
| bytes32 toHexString | 287.00 ns/op | 287.00 ns/op | 1.00 |
| bytes32 Buffer.toString(hex) | 172.00 ns/op | 165.00 ns/op | 1.04 |
| bytes32 Buffer.toString(hex) from Uint8Array | 242.00 ns/op | 234.00 ns/op | 1.03 |
| bytes32 Buffer.toString(hex) + 0x | 171.00 ns/op | 159.00 ns/op | 1.08 |
| Return object 10000 times | 0.21110 ns/op | 0.21370 ns/op | 0.99 |
| Throw Error 10000 times | 3.3786 us/op | 3.3065 us/op | 1.02 |
| toHex | 96.228 ns/op | 88.527 ns/op | 1.09 |
| Buffer.from | 83.394 ns/op | 86.095 ns/op | 0.97 |
| shared Buffer | 53.178 ns/op | 52.477 ns/op | 1.01 |
| fastMsgIdFn sha256 / 200 bytes | 1.4950 us/op | 1.4730 us/op | 1.01 |
| fastMsgIdFn h32 xxhash / 200 bytes | 172.00 ns/op | 158.00 ns/op | 1.09 |
| fastMsgIdFn h64 xxhash / 200 bytes | 221.00 ns/op | 205.00 ns/op | 1.08 |
| fastMsgIdFn sha256 / 1000 bytes | 4.7810 us/op | 4.8000 us/op | 1.00 |
| fastMsgIdFn h32 xxhash / 1000 bytes | 261.00 ns/op | 249.00 ns/op | 1.05 |
| fastMsgIdFn h64 xxhash / 1000 bytes | 270.00 ns/op | 249.00 ns/op | 1.08 |
| fastMsgIdFn sha256 / 10000 bytes | 42.671 us/op | 43.298 us/op | 0.99 |
| fastMsgIdFn h32 xxhash / 10000 bytes | 1.2940 us/op | 1.2640 us/op | 1.02 |
| fastMsgIdFn h64 xxhash / 10000 bytes | 845.00 ns/op | 804.00 ns/op | 1.05 |
| send data - 1000 256B messages | 4.8999 ms/op | 4.9460 ms/op | 0.99 |
| send data - 1000 512B messages | 4.5865 ms/op | 4.9954 ms/op | 0.92 |
| send data - 1000 1024B messages | 5.7566 ms/op | 5.1021 ms/op | 1.13 |
| send data - 1000 1200B messages | 5.3470 ms/op | 5.5398 ms/op | 0.97 |
| send data - 1000 2048B messages | 5.5920 ms/op | 6.1396 ms/op | 0.91 |
| send data - 1000 4096B messages | 6.2755 ms/op | 7.0511 ms/op | 0.89 |
| send data - 1000 16384B messages | 71.664 ms/op | 22.447 ms/op | 3.19 |
| send data - 1000 65536B messages | 243.03 ms/op | 120.61 ms/op | 2.01 |
| enrSubnets - fastDeserialize 64 bits | 740.00 ns/op | 733.00 ns/op | 1.01 |
| enrSubnets - ssz BitVector 64 bits | 263.00 ns/op | 251.00 ns/op | 1.05 |
| enrSubnets - fastDeserialize 4 bits | 105.00 ns/op | 102.00 ns/op | 1.03 |
| enrSubnets - ssz BitVector 4 bits | 254.00 ns/op | 247.00 ns/op | 1.03 |
| prioritizePeers score -10:0 att 32-0.1 sync 2-0 | 199.33 us/op | 207.10 us/op | 0.96 |
| prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 | 243.08 us/op | 425.28 us/op | 0.57 |
| prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 | 341.67 us/op | 358.49 us/op | 0.95 |
| prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 | 600.28 us/op | 611.20 us/op | 0.98 |
| prioritizePeers score 0:0 att 64-1 sync 4-1 | 699.43 us/op | 695.35 us/op | 1.01 |
| array of 16000 items push then shift | 1.2633 us/op | 1.2197 us/op | 1.04 |
| LinkedList of 16000 items push then shift | 7.5950 ns/op | 7.5140 ns/op | 1.01 |
| array of 16000 items push then pop | 74.641 ns/op | 67.554 ns/op | 1.10 |
| LinkedList of 16000 items push then pop | 6.1210 ns/op | 5.9840 ns/op | 1.02 |
| array of 24000 items push then shift | 1.8758 us/op | 1.7932 us/op | 1.05 |
| LinkedList of 24000 items push then shift | 7.5490 ns/op | 7.4560 ns/op | 1.01 |
| array of 24000 items push then pop | 108.91 ns/op | 98.161 ns/op | 1.11 |
| LinkedList of 24000 items push then pop | 6.4220 ns/op | 6.4230 ns/op | 1.00 |
| intersect bitArray bitLen 8 | 4.9270 ns/op | 4.6700 ns/op | 1.06 |
| intersect array and set length 8 | 30.282 ns/op | 28.516 ns/op | 1.06 |
| intersect bitArray bitLen 128 | 24.434 ns/op | 23.160 ns/op | 1.06 |
| intersect array and set length 128 | 507.20 ns/op | 487.02 ns/op | 1.04 |
| bitArray.getTrueBitIndexes() bitLen 128 | 1.0590 us/op | 1.0110 us/op | 1.05 |
| bitArray.getTrueBitIndexes() bitLen 248 | 1.8240 us/op | 1.7400 us/op | 1.05 |
| bitArray.getTrueBitIndexes() bitLen 512 | 3.7300 us/op | 3.5580 us/op | 1.05 |
| Full columns - reconstruct all 6 blobs | 167.15 us/op | 218.00 us/op | 0.77 |
| Full columns - reconstruct half of the blobs out of 6 | 75.603 us/op | 104.45 us/op | 0.72 |
| Full columns - reconstruct single blob out of 6 | 31.131 us/op | 41.269 us/op | 0.75 |
| Half columns - reconstruct all 6 blobs | 384.40 ms/op | 380.79 ms/op | 1.01 |
| Half columns - reconstruct half of the blobs out of 6 | 190.01 ms/op | 190.13 ms/op | 1.00 |
| Half columns - reconstruct single blob out of 6 | 67.672 ms/op | 68.187 ms/op | 0.99 |
| Full columns - reconstruct all 10 blobs | 273.28 us/op | 298.97 us/op | 0.91 |
| Full columns - reconstruct half of the blobs out of 10 | 134.41 us/op | 199.80 us/op | 0.67 |
| Full columns - reconstruct single blob out of 10 | 31.760 us/op | 60.783 us/op | 0.52 |
| Half columns - reconstruct all 10 blobs | 664.14 ms/op | 646.12 ms/op | 1.03 |
| Half columns - reconstruct half of the blobs out of 10 | 327.95 ms/op | 325.10 ms/op | 1.01 |
| Half columns - reconstruct single blob out of 10 | 67.753 ms/op | 71.936 ms/op | 0.94 |
| Full columns - reconstruct all 20 blobs | 1.5123 ms/op | 600.25 us/op | 2.52 |
| Full columns - reconstruct half of the blobs out of 20 | 182.83 us/op | 342.49 us/op | 0.53 |
| Full columns - reconstruct single blob out of 20 | 29.014 us/op | 30.670 us/op | 0.95 |
| Half columns - reconstruct all 20 blobs | 1.2665 s/op | 1.2549 s/op | 1.01 |
| Half columns - reconstruct half of the blobs out of 20 | 647.86 ms/op | 632.71 ms/op | 1.02 |
| Half columns - reconstruct single blob out of 20 | 69.583 ms/op | 66.834 ms/op | 1.04 |
| Set add up to 64 items then delete first | 2.0843 us/op | 1.9740 us/op | 1.06 |
| OrderedSet add up to 64 items then delete first | 3.2997 us/op | 3.1543 us/op | 1.05 |
| Set add up to 64 items then delete last | 2.1124 us/op | 2.0294 us/op | 1.04 |
| OrderedSet add up to 64 items then delete last | 3.6456 us/op | 3.2489 us/op | 1.12 |
| Set add up to 64 items then delete middle | 2.5827 us/op | 2.0964 us/op | 1.23 |
| OrderedSet add up to 64 items then delete middle | 5.6265 us/op | 4.7924 us/op | 1.17 |
| Set add up to 128 items then delete first | 4.4057 us/op | 4.1395 us/op | 1.06 |
| OrderedSet add up to 128 items then delete first | 6.8027 us/op | 5.8876 us/op | 1.16 |
| Set add up to 128 items then delete last | 4.2770 us/op | 3.9331 us/op | 1.09 |
| OrderedSet add up to 128 items then delete last | 6.9134 us/op | 7.6562 us/op | 0.90 |
| Set add up to 128 items then delete middle | 4.8515 us/op | 4.8757 us/op | 1.00 |
| OrderedSet add up to 128 items then delete middle | 14.620 us/op | 14.240 us/op | 1.03 |
| Set add up to 256 items then delete first | 9.2509 us/op | 9.3791 us/op | 0.99 |
| OrderedSet add up to 256 items then delete first | 13.692 us/op | 16.319 us/op | 0.84 |
| Set add up to 256 items then delete last | 8.9046 us/op | 10.262 us/op | 0.87 |
| OrderedSet add up to 256 items then delete last | 14.469 us/op | 14.088 us/op | 1.03 |
| Set add up to 256 items then delete middle | 9.0376 us/op | 8.0674 us/op | 1.12 |
| OrderedSet add up to 256 items then delete middle | 42.060 us/op | 39.805 us/op | 1.06 |
| pass gossip attestations to forkchoice per slot | 2.6828 ms/op | 2.4974 ms/op | 1.07 |
| forkChoice updateHead vc 100000 bc 64 eq 0 | 403.46 us/op | 389.86 us/op | 1.03 |
| forkChoice updateHead vc 600000 bc 64 eq 0 | 2.4794 ms/op | 2.2227 ms/op | 1.12 |
| forkChoice updateHead vc 1000000 bc 64 eq 0 | 3.9534 ms/op | 3.7068 ms/op | 1.07 |
| forkChoice updateHead vc 600000 bc 320 eq 0 | 2.3283 ms/op | 2.2506 ms/op | 1.03 |
| forkChoice updateHead vc 600000 bc 1200 eq 0 | 2.4865 ms/op | 2.3640 ms/op | 1.05 |
| forkChoice updateHead vc 600000 bc 7200 eq 0 | 3.8660 ms/op | 3.3149 ms/op | 1.17 |
| forkChoice updateHead vc 600000 bc 64 eq 1000 | 3.0401 ms/op | 2.7823 ms/op | 1.09 |
| forkChoice updateHead vc 600000 bc 64 eq 10000 | 3.1507 ms/op | 3.0262 ms/op | 1.04 |
| forkChoice updateHead vc 600000 bc 64 eq 300000 | 7.8718 ms/op | 7.0111 ms/op | 1.12 |
| computeDeltas 1400000 validators 0% inactive | 12.970 ms/op | 12.515 ms/op | 1.04 |
| computeDeltas 1400000 validators 10% inactive | 12.232 ms/op | 11.812 ms/op | 1.04 |
| computeDeltas 1400000 validators 20% inactive | 11.007 ms/op | 10.596 ms/op | 1.04 |
| computeDeltas 1400000 validators 50% inactive | 8.6916 ms/op | 8.0472 ms/op | 1.08 |
| computeDeltas 2100000 validators 0% inactive | 19.477 ms/op | 18.706 ms/op | 1.04 |
| computeDeltas 2100000 validators 10% inactive | 18.622 ms/op | 17.803 ms/op | 1.05 |
| computeDeltas 2100000 validators 20% inactive | 16.708 ms/op | 16.165 ms/op | 1.03 |
| computeDeltas 2100000 validators 50% inactive | 12.941 ms/op | 9.5266 ms/op | 1.36 |
| altair processAttestation - 250000 vs - 7PWei normalcase | 2.9898 ms/op | 2.4686 ms/op | 1.21 |
| altair processAttestation - 250000 vs - 7PWei worstcase | 3.5929 ms/op | 3.5976 ms/op | 1.00 |
| altair processAttestation - setStatus - 1/6 committees join | 108.06 us/op | 96.782 us/op | 1.12 |
| altair processAttestation - setStatus - 1/3 committees join | 204.84 us/op | 219.50 us/op | 0.93 |
| altair processAttestation - setStatus - 1/2 committees join | 295.39 us/op | 281.15 us/op | 1.05 |
| altair processAttestation - setStatus - 2/3 committees join | 380.04 us/op | 355.96 us/op | 1.07 |
| altair processAttestation - setStatus - 4/5 committees join | 540.86 us/op | 540.14 us/op | 1.00 |
| altair processAttestation - setStatus - 100% committees join | 681.08 us/op | 613.75 us/op | 1.11 |
| altair processBlock - 250000 vs - 7PWei normalcase | 4.8825 ms/op | 4.4057 ms/op | 1.11 |
| altair processBlock - 250000 vs - 7PWei normalcase hashState | 15.733 ms/op | 17.246 ms/op | 0.91 |
| altair processBlock - 250000 vs - 7PWei worstcase | 23.619 ms/op | 22.803 ms/op | 1.04 |
| altair processBlock - 250000 vs - 7PWei worstcase hashState | 43.467 ms/op | 42.375 ms/op | 1.03 |
| phase0 processBlock - 250000 vs - 7PWei normalcase | 1.5847 ms/op | 1.4336 ms/op | 1.11 |
| phase0 processBlock - 250000 vs - 7PWei worstcase | 21.709 ms/op | 17.429 ms/op | 1.25 |
| altair processEth1Data - 250000 vs - 7PWei normalcase | 315.05 us/op | 275.82 us/op | 1.14 |
| getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 | 6.2160 us/op | 3.0100 us/op | 2.07 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 | 22.686 us/op | 18.765 us/op | 1.21 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 | 5.8660 us/op | 5.6920 us/op | 1.03 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 | 4.2690 us/op | 4.4060 us/op | 0.97 |
| getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 | 96.638 us/op | 86.826 us/op | 1.11 |
| getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 | 1.4049 ms/op | 1.4322 ms/op | 0.98 |
| getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 | 1.8755 ms/op | 1.8114 ms/op | 1.04 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 | 1.8377 ms/op | 1.7985 ms/op | 1.02 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 | 5.2363 ms/op | 4.5213 ms/op | 1.16 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 | 2.1165 ms/op | 2.0123 ms/op | 1.05 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 | 4.5913 ms/op | 3.7384 ms/op | 1.23 |
| Tree 40 250000 create | 373.02 ms/op | 335.87 ms/op | 1.11 |
| Tree 40 250000 get(125000) | 96.120 ns/op | 88.860 ns/op | 1.08 |
| Tree 40 250000 set(125000) | 1.0392 us/op | 967.62 ns/op | 1.07 |
| Tree 40 250000 toArray() | 19.300 ms/op | 16.364 ms/op | 1.18 |
| Tree 40 250000 iterate all - toArray() + loop | 17.350 ms/op | 21.474 ms/op | 0.81 |
| Tree 40 250000 iterate all - get(i) | 42.321 ms/op | 42.052 ms/op | 1.01 |
| Array 250000 create | 2.3416 ms/op | 2.3758 ms/op | 0.99 |
| Array 250000 clone - spread | 713.04 us/op | 716.48 us/op | 1.00 |
| Array 250000 get(125000) | 0.30500 ns/op | 0.29900 ns/op | 1.02 |
| Array 250000 set(125000) | 0.30400 ns/op | 0.30400 ns/op | 1.00 |
| Array 250000 iterate all - loop | 59.106 us/op | 57.676 us/op | 1.02 |
| phase0 afterProcessEpoch - 250000 vs - 7PWei | 46.122 ms/op | 47.557 ms/op | 0.97 |
| Array.fill - length 1000000 | 2.3569 ms/op | 2.9463 ms/op | 0.80 |
| Array push - length 1000000 | 9.9127 ms/op | 10.147 ms/op | 0.98 |
| Array.get | 0.21133 ns/op | 0.20565 ns/op | 1.03 |
| Uint8Array.get | 0.24475 ns/op | 0.23343 ns/op | 1.05 |
| phase0 beforeProcessEpoch - 250000 vs - 7PWei | 29.665 ms/op | 17.140 ms/op | 1.73 |
| altair processEpoch - mainnet_e81889 | 326.79 ms/op | 267.08 ms/op | 1.22 |
| mainnet_e81889 - altair beforeProcessEpoch | 38.728 ms/op | 18.222 ms/op | 2.13 |
| mainnet_e81889 - altair processJustificationAndFinalization | 6.8790 us/op | 7.1760 us/op | 0.96 |
| mainnet_e81889 - altair processInactivityUpdates | 6.9550 ms/op | 7.1201 ms/op | 0.98 |
| mainnet_e81889 - altair processRewardsAndPenalties | 22.185 ms/op | 22.439 ms/op | 0.99 |
| mainnet_e81889 - altair processRegistryUpdates | 575.00 ns/op | 550.00 ns/op | 1.05 |
| mainnet_e81889 - altair processSlashings | 138.00 ns/op | 146.00 ns/op | 0.95 |
| mainnet_e81889 - altair processEth1DataReset | 136.00 ns/op | 136.00 ns/op | 1.00 |
| mainnet_e81889 - altair processEffectiveBalanceUpdates | 5.4254 ms/op | 6.0572 ms/op | 0.90 |
| mainnet_e81889 - altair processSlashingsReset | 718.00 ns/op | 683.00 ns/op | 1.05 |
| mainnet_e81889 - altair processRandaoMixesReset | 1.4980 us/op | 1.3390 us/op | 1.12 |
| mainnet_e81889 - altair processHistoricalRootsUpdate | 137.00 ns/op | 132.00 ns/op | 1.04 |
| mainnet_e81889 - altair processParticipationFlagUpdates | 477.00 ns/op | 444.00 ns/op | 1.07 |
| mainnet_e81889 - altair processSyncCommitteeUpdates | 111.00 ns/op | 112.00 ns/op | 0.99 |
| mainnet_e81889 - altair afterProcessEpoch | 42.932 ms/op | 41.694 ms/op | 1.03 |
| capella processEpoch - mainnet_e217614 | 908.45 ms/op | 764.40 ms/op | 1.19 |
| mainnet_e217614 - capella beforeProcessEpoch | 59.453 ms/op | 55.066 ms/op | 1.08 |
| mainnet_e217614 - capella processJustificationAndFinalization | 6.6600 us/op | 5.4190 us/op | 1.23 |
| mainnet_e217614 - capella processInactivityUpdates | 14.952 ms/op | 14.346 ms/op | 1.04 |
| mainnet_e217614 - capella processRewardsAndPenalties | 88.725 ms/op | 84.333 ms/op | 1.05 |
| mainnet_e217614 - capella processRegistryUpdates | 4.5500 us/op | 4.3540 us/op | 1.05 |
| mainnet_e217614 - capella processSlashings | 134.00 ns/op | 127.00 ns/op | 1.06 |
| mainnet_e217614 - capella processEth1DataReset | 136.00 ns/op | 127.00 ns/op | 1.07 |
| mainnet_e217614 - capella processEffectiveBalanceUpdates | 24.671 ms/op | 11.857 ms/op | 2.08 |
| mainnet_e217614 - capella processSlashingsReset | 710.00 ns/op | 653.00 ns/op | 1.09 |
| mainnet_e217614 - capella processRandaoMixesReset | 1.2040 us/op | 1.0880 us/op | 1.11 |
| mainnet_e217614 - capella processHistoricalRootsUpdate | 134.00 ns/op | 125.00 ns/op | 1.07 |
| mainnet_e217614 - capella processParticipationFlagUpdates | 447.00 ns/op | 408.00 ns/op | 1.10 |
| mainnet_e217614 - capella afterProcessEpoch | 106.78 ms/op | 105.29 ms/op | 1.01 |
| phase0 processEpoch - mainnet_e58758 | 309.94 ms/op | 289.13 ms/op | 1.07 |
| mainnet_e58758 - phase0 beforeProcessEpoch | 65.327 ms/op | 60.324 ms/op | 1.08 |
| mainnet_e58758 - phase0 processJustificationAndFinalization | 6.3520 us/op | 5.6000 us/op | 1.13 |
| mainnet_e58758 - phase0 processRewardsAndPenalties | 15.947 ms/op | 15.269 ms/op | 1.04 |
| mainnet_e58758 - phase0 processRegistryUpdates | 2.2960 us/op | 2.1750 us/op | 1.06 |
| mainnet_e58758 - phase0 processSlashings | 134.00 ns/op | 130.00 ns/op | 1.03 |
| mainnet_e58758 - phase0 processEth1DataReset | 266.00 ns/op | 259.00 ns/op | 1.03 |
| mainnet_e58758 - phase0 processEffectiveBalanceUpdates | 2.6389 ms/op | 800.18 us/op | 3.30 |
| mainnet_e58758 - phase0 processSlashingsReset | 960.00 ns/op | 822.00 ns/op | 1.17 |
| mainnet_e58758 - phase0 processRandaoMixesReset | 1.4140 us/op | 1.1910 us/op | 1.19 |
| mainnet_e58758 - phase0 processHistoricalRootsUpdate | 143.00 ns/op | 126.00 ns/op | 1.13 |
| mainnet_e58758 - phase0 processParticipationRecordUpdates | 1.2190 us/op | 951.00 ns/op | 1.28 |
| mainnet_e58758 - phase0 afterProcessEpoch | 33.869 ms/op | 32.005 ms/op | 1.06 |
| phase0 processEffectiveBalanceUpdates - 250000 normalcase | 1.0174 ms/op | 947.75 us/op | 1.07 |
| phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 | 3.5136 ms/op | 1.9485 ms/op | 1.80 |
| altair processInactivityUpdates - 250000 normalcase | 13.199 ms/op | 10.327 ms/op | 1.28 |
| altair processInactivityUpdates - 250000 worstcase | 13.713 ms/op | 10.192 ms/op | 1.35 |
| phase0 processRegistryUpdates - 250000 normalcase | 2.4560 us/op | 2.1810 us/op | 1.13 |
| phase0 processRegistryUpdates - 250000 badcase_full_deposits | 141.20 us/op | 130.04 us/op | 1.09 |
| phase0 processRegistryUpdates - 250000 worstcase 0.5 | 65.825 ms/op | 59.517 ms/op | 1.11 |
| altair processRewardsAndPenalties - 250000 normalcase | 15.966 ms/op | 16.003 ms/op | 1.00 |
| altair processRewardsAndPenalties - 250000 worstcase | 16.355 ms/op | 15.256 ms/op | 1.07 |
| phase0 getAttestationDeltas - 250000 normalcase | 7.7238 ms/op | 5.0876 ms/op | 1.52 |
| phase0 getAttestationDeltas - 250000 worstcase | 11.429 ms/op | 7.9911 ms/op | 1.43 |
| phase0 processSlashings - 250000 worstcase | 62.305 us/op | 56.276 us/op | 1.11 |
| altair processSyncCommitteeUpdates - 250000 | 11.570 ms/op | 9.5739 ms/op | 1.21 |
| BeaconState.hashTreeRoot - No change | 172.00 ns/op | 172.00 ns/op | 1.00 |
| BeaconState.hashTreeRoot - 1 full validator | 86.779 us/op | 78.398 us/op | 1.11 |
| BeaconState.hashTreeRoot - 32 full validator | 878.50 us/op | 819.77 us/op | 1.07 |
| BeaconState.hashTreeRoot - 512 full validator | 9.0960 ms/op | 8.7617 ms/op | 1.04 |
| BeaconState.hashTreeRoot - 1 validator.effectiveBalance | 113.29 us/op | 95.440 us/op | 1.19 |
| BeaconState.hashTreeRoot - 32 validator.effectiveBalance | 1.4809 ms/op | 1.3696 ms/op | 1.08 |
| BeaconState.hashTreeRoot - 512 validator.effectiveBalance | 20.302 ms/op | 17.029 ms/op | 1.19 |
| BeaconState.hashTreeRoot - 1 balances | 88.288 us/op | 70.368 us/op | 1.25 |
| BeaconState.hashTreeRoot - 32 balances | 800.06 us/op | 700.26 us/op | 1.14 |
| BeaconState.hashTreeRoot - 512 balances | 7.0952 ms/op | 5.9649 ms/op | 1.19 |
| BeaconState.hashTreeRoot - 250000 balances | 136.07 ms/op | 126.22 ms/op | 1.08 |
| aggregationBits - 2048 els - zipIndexesInBitList | 19.715 us/op | 18.314 us/op | 1.08 |
| regular array get 100000 times | 23.117 us/op | 21.951 us/op | 1.05 |
| wrappedArray get 100000 times | 23.304 us/op | 22.112 us/op | 1.05 |
| arrayWithProxy get 100000 times | 10.257 ms/op | 16.791 ms/op | 0.61 |
| ssz.Root.equals | 21.654 ns/op | 20.547 ns/op | 1.05 |
| byteArrayEquals | 21.450 ns/op | 20.422 ns/op | 1.05 |
| Buffer.compare | 8.9300 ns/op | 8.4600 ns/op | 1.06 |
| processSlot - 1 slots | 10.348 us/op | 8.6800 us/op | 1.19 |
| processSlot - 32 slots | 2.1896 ms/op | 1.8763 ms/op | 1.17 |
| getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei | 3.5847 ms/op | 3.5573 ms/op | 1.01 |
| getCommitteeAssignments - req 1 vs - 250000 vc | 1.6312 ms/op | 1.5898 ms/op | 1.03 |
| getCommitteeAssignments - req 100 vs - 250000 vc | 3.3718 ms/op | 3.2733 ms/op | 1.03 |
| getCommitteeAssignments - req 1000 vs - 250000 vc | 3.6045 ms/op | 3.4983 ms/op | 1.03 |
| findModifiedValidators - 10000 modified validators | 733.80 ms/op | 671.22 ms/op | 1.09 |
| findModifiedValidators - 1000 modified validators | 523.21 ms/op | 392.96 ms/op | 1.33 |
| findModifiedValidators - 100 modified validators | 290.99 ms/op | 275.43 ms/op | 1.06 |
| findModifiedValidators - 10 modified validators | 232.99 ms/op | 195.69 ms/op | 1.19 |
| findModifiedValidators - 1 modified validators | 127.33 ms/op | 141.65 ms/op | 0.90 |
| findModifiedValidators - no difference | 150.60 ms/op | 152.58 ms/op | 0.99 |
| migrate state 1500000 validators, 3400 modified, 2000 new | 3.2050 s/op | 2.6225 s/op | 1.22 |
| RootCache.getBlockRootAtSlot - 250000 vs - 7PWei | 3.8000 ns/op | 3.6400 ns/op | 1.04 |
| state getBlockRootAtSlot - 250000 vs - 7PWei | 427.00 ns/op | 330.75 ns/op | 1.29 |
| computeProposerIndex 100000 validators | 1.4170 ms/op | 1.2639 ms/op | 1.12 |
| getNextSyncCommitteeIndices 1000 validators | 2.9468 ms/op | 2.7579 ms/op | 1.07 |
| getNextSyncCommitteeIndices 10000 validators | 26.154 ms/op | 24.135 ms/op | 1.08 |
| getNextSyncCommitteeIndices 100000 validators | 93.251 ms/op | 79.462 ms/op | 1.17 |
| computeProposers - vc 250000 | 575.98 us/op | 552.18 us/op | 1.04 |
| computeEpochShuffling - vc 250000 | 40.903 ms/op | 38.116 ms/op | 1.07 |
| getNextSyncCommittee - vc 250000 | 12.078 ms/op | 9.1994 ms/op | 1.31 |
| nodejs block root to RootHex using toHex | 94.814 ns/op | 84.136 ns/op | 1.13 |
| nodejs block root to RootHex using toRootHex | 54.773 ns/op | 49.338 ns/op | 1.11 |
| nodejs fromHex(blob) | 1.0713 ms/op | 680.49 us/op | 1.57 |
| nodejs fromHexInto(blob) | 631.89 us/op | 599.34 us/op | 1.05 |
| nodejs block root to RootHex using the deprecated toHexString | 486.40 ns/op | 442.91 ns/op | 1.10 |
| nodejs byteArrayEquals 32 bytes (block root) | 25.942 ns/op | 24.900 ns/op | 1.04 |
| nodejs byteArrayEquals 48 bytes (pubkey) | 38.107 ns/op | 35.695 ns/op | 1.07 |
| nodejs byteArrayEquals 96 bytes (signature) | 34.167 ns/op | 31.936 ns/op | 1.07 |
| nodejs byteArrayEquals 1024 bytes | 41.089 ns/op | 39.579 ns/op | 1.04 |
| nodejs byteArrayEquals 131072 bytes (blob) | 1.8210 us/op | 1.6856 us/op | 1.08 |
| browser block root to RootHex using toHex | 154.53 ns/op | 137.62 ns/op | 1.12 |
| browser block root to RootHex using toRootHex | 134.26 ns/op | 124.72 ns/op | 1.08 |
| browser fromHex(blob) | 1.8290 ms/op | 1.4143 ms/op | 1.29 |
| browser fromHexInto(blob) | 653.49 us/op | 598.15 us/op | 1.09 |
| browser block root to RootHex using the deprecated toHexString | 363.74 ns/op | 312.36 ns/op | 1.16 |
| browser byteArrayEquals 32 bytes (block root) | 29.182 ns/op | 26.673 ns/op | 1.09 |
| browser byteArrayEquals 48 bytes (pubkey) | 41.303 ns/op | 37.484 ns/op | 1.10 |
| browser byteArrayEquals 96 bytes (signature) | 77.196 ns/op | 70.744 ns/op | 1.09 |
| browser byteArrayEquals 1024 bytes | 792.27 ns/op | 724.60 ns/op | 1.09 |
| browser byteArrayEquals 131072 bytes (blob) | 101.13 us/op | 90.996 us/op | 1.11 |
by benchmarkbot/action
twoeths
left a comment
There was a problem hiding this comment.
- the fix of #8150 was to throw error if there is 0-block returned, this is incorrect in case the network really has 0 block, especially in devnets. But it rarely happens and the fix helped us to be able to range sync on fulu devnets at the time
- this fix handles 0-block returned, but there could be issue in regular networks where peers did not return correctly, at least I know lodestar did not handle it well
the problem with #8147 is that node downloaded multiple empty block epoch and got stuck
it happened there in fulu devnets because earliestAvailableSlot was not fully implemented by clients
the change seems good to me but we need to make sure we can range sync on the next glamsterdam-devnet-5 + hoodi + mainnet
Motivation
A peer's correct response for an epoch in which every slot was skipped is zero
blocks. Range sync rejected that response, so the node could not sync past an
empty epoch during periods of poor chain liveness.
Two interacting defects, both introduced during the fulu/PeerDAS sync work:
validateBlockByRangeResponse threw MISSING_BLOCKS_RESPONSE on a 0-block
response. In sendBatch this became a downloadingError; since every peer
returns empty for a genuinely empty epoch, all retries failed and after
MAX_BATCH_DOWNLOAD_ATTEMPTS the batch threw MAX_DOWNLOAD_ATTEMPTS and the
whole SyncChain errored out. A second copy of the throw in validateResponses
fired for post-Deneb epochs that also carry a blobs/columns request.
The MAX_LOOK_AHEAD_EPOCHS guard in includeNextBatch counted batches in the
AwaitingValidation state. Empty batches reach AwaitingValidation but never
call advanceChain (only non-empty batches do), so lastEpochWithProcessBlocks
never advances. A run of empty epochs longer than MAX_LOOK_AHEAD_EPOCHS (2)
filled the look-ahead window, includeNextBatch returned null, and the chain
stalled with nothing left to download or process. This deadlock is why the
naive "return [] instead of throw" fix had previously been reverted.
Description
MISSING_BLOCKS_RESPONSE warning instead of throwing. Whether an epoch is
genuinely empty is enforced at the chain level: the empty batch is held in
AwaitingValidation and only confirmed once a later batch imports a block, so
a peer cannot stall sync by falsely claiming an epoch is empty.
the data request's slot range (parent-by-root columns and envelopes still
run).
mirroring the BATCH_BUFFER_SIZE carve-out.
Archeology
2025-09-18) -- the fulu block-input/PeerDAS sync rewrite that deleted the old
Deneb-era beaconBlocksMaybeBlobsByRange.ts downloader, which had tolerated
empty epochs. The empty-blocks throw was then cemented in fix: refactor validateColumnsByRangeResponse #8482 (fix:
refactor validateColumnsByRangeResponse, 2025-10-27, b992c32).
PeerDAS sync work. The original 2021 BATCH_BUFFER_SIZE check directly above it
(f46b63f, dapplion) deliberately excluded AwaitingValidation batches "to
prevent stalling sync if the current processing window is contained in a long
range of skip slots." The new guard did not replicate that carve-out,
re-creating the exact stall the 2021 code was written to avoid.
AI Assistance Disclosure